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ABSTRACT 

Wc describe the selection of a sample of photometrically-defined Lyman break 
galaxies (LBGs) at z 5 using the multi-wavclcngth imaging data of the ESO Remote 
Galaxy Survey (ERGS). The data is drawn from ten widely-separated fields covering 
a total sky area of 275 arcmin^. Starting with a simple colour {R — I > 1.3) and 
magnitude (/ < 26.3) cut to isolate the Lyman break and then refining the sample by 
applying further optical and near-infrared photometric criteria we identify a sample 
of 253 LEG candidates. We carefully model the completeness of this sample and the 
factors that affect its reliability. There is considerable overlap between this sample 
and a spectroscopically-confirmed sample drawn from the same survey and this allows 
us to determine the reliability of the optical photometric selection (~ 60 per cent) 
and to show that the reliability can be significantly improved (to ~ 80 per cent) by 
applying near-infrared waveband criteria to exclude very red contaminants. Even this 
high level of reliability may compromise some statistical studies of LBG properties. 
We show that over 30 per cent of the highest reliability candidates have multiple 
UV- luminous components and/or disturbed morphology in HST imaging, though it 
is unclear whether this represents multiple interacting/merging sources or individual 
large sources with multiple UV bright regions. Using this sample we confirm that the 
normalisation of the bright end of the z = 5 UV luminosity function (down to M*) 
is lower than the same at z = 4 by a factor of 3. Using a Schechter fit we determine 
M^y = —20.9 ± 0.2. We discuss whether it is reasonable to expect the UV luminosity 
function to follow a Schechter function, given the UV emission is short-lived and 
stochastic, and does not necessarily trace the underlying mass of the galaxy. 
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, 1 INTRODUCTION 

Until the turn of the century, very few spectroscopically- 
confirmed galaxies at z ~ 5 and above had been identified 
and fewer still studied in any detail. Given that this redshift 
range covers the first billion years after the Big Bang, this 
meant that there was little concrete evidence of how early 
galaxy formation proceeds. With the advent of deep imaging 
with 8m ground-based telescopes, and with the GOODS and 
HUDF projects carried out with HST, it has become possible 
to photometrically identify distant galaxy candidates and 
potentially confirm them through spectroscopy. 

By extending the Lyman break techniqu e 
(iGuhathakurta et al ] |l990l : ISteidel fc Hamihonll 19911 1993h 



is now possible to securely identify multiple high redshift 
galaxies in individual 8m telescope pointings. These Lyman 
Break Galaxies (LBGs) are identified via their redshifted 
UV continuum emission arising from strong unobscured 
star formation within the systems. Subsequently, similar 
studies have been carried out by other author s in order to 
select galaxies with redshifts above z = 4.5 JOuchi et al.l 
2004h and p otentially to 2 > 6 (e.g. IStanwav et all l2003l . 
20041 . l2008al ). 



to z ~ 5 using R, I and z— band imaging. iLehnert fc Bremer! 
(|2003l ) showed that with the right observational setup it 



While some of these studies have included spectro- 
scopic confirmation of samples derived from photometry, 
many others have not. These have relied on the high success 
rate of spectroscopic co nfirmation in previous work ( e.g. 
iLehnert fc Bremeillioosl . confirmed six of a sample of twelve 
candidates in a 40 arcmin^ field to be at 4.8 < z < 5.8) to ar- 
gue that photometric selection was reliable enough to carry 
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out statist ical studies on photom etric samples. However, as 
shown by IStanwav etHI (HooiH), the reliability and com- 
pleteness of such samples depends on a complex combination 
of factors, particularly the filter set and CCD response used 
to carry out the imaging. In the absence of spectroscopy, 
other data is required to help exclude contaminating galactic 
stars and z ^ 1 elliptical galaxies from photometric samples, 
such as HST and/or deep near-IR ground-based imaging. 
One of the main advantages of purely photometric samples 
is that they can cover a relatively large area of sky in com- 
parison to spectroscopic samples. However, given the diffi- 
culty of obtaining HST and near-IR data of sufficient depth 
over a large sky area, the use of photometric samples are 
often compromised. 

For this reason, despite the apparent ease of defining a 
photometric sample, there are distinct advantages in using 
a sample which has clear spectroscopic evidence of reliabil- 
ity when attempting to understand the statistics and other 
properties of star forming galaxies at 2 > 5. Accurate deter- 
minations of their clustering statistics, luminosity functions, 
star formation histories, stellar population properties and 
contribution to reionization all require an understanding of 
the contaminants in the sample being used for the study. 
Adding to this the need for accurate redshifts in order to 
carry out many detailed studies of individual high redshift 
systems, the need for a well-defined photometric sample of 
z ^ 5 LBGs with good spectroscopic follow-up is clear. 

iLehnert fc BremeJ (|2003h showed that it was possible to 
obtain ground-based samples of such objects with a reason- 
able level of spectroscopic confirmation. However, the study 
covered a comparatively small area of sky and so could only 
be used to determine the simplest of statistical properties for 
such sources. In more recent years larg er areas have been 
cove red by spectro s copic programs e.g. lAndo et all (|2007l l 
and IVanzella et al.l (j2009( ) which confirmed 10 and 32 high 
redshift galaxies respectively. Although these surveys are im- 
portant to the understanding of the selection of high red- 
shift galaxies and their properties, they often only target 
the brightest candidates spectroscopically or do not uni- 
formly probe the corresponding photometric sample. One 
of the goals of this survey, described here and in future pa- 
pers, is to observe many high redshift LEG candidates over 
a range of luminosities and colours to discover if the rela- 
tively high completeness seen in previous studies is applica- 
ble at fainter magnitudes. Although the level of contamina- 
tion of low re dshift Lyman break galaxy samples are often 
very low, e.g. ISteidel et all (|2003h . such a low level cannot 
be assumed for higher redshift samples. The differences in 
the selections functions and colours used mean that the two 
samples are contaminated by completely different galactic 
and extragalactic populations. 

To address this, and other, goals we have carried out 
a large programme of imaging and spectroscopic observa- 
tions which cover a sky area f ourtee n times larger than 
the original iLehnert fc Bremeij (|2003l l study including an 
exp anded region of 160 ar cmin'^ centered on the orig- 
inal ILehnert fc Bremeil l|2003h pointing. This paper intro- 
duces the imaging data sets used to generate a sample of 
2 ~ 5 LBG candidates which were foUo wed-up by the same 
spectr oscopic setup used in the original ILehnert fc Bremen 
(|2003h work. Also it describes a photometric sample of z ~ 5 
LBG candidates with well-determined reliability and com- 



pleteness which can be used for statistical study, such as 
a determination and discussion of the z ^ 5 LBG UV lu- 
minosity function presented here and a brief comparison of 
the results of a single R — I colour cut to that of a sample 
selection with an additional I — z criterion. 

The data presented here are drawn from from ten ~ 40 
arcmin^ fields widely-separated on the sky. Observations 
were carried out as part of an ESO Large Programme, "The 
ESO Remote Galaxy Survey", or ERGS (PI M. Bremer, 
ID 175.A-0706). This capitalised on existing deep optical 
imaging of multiple fields already obtai ned by the ESO Dis - 
tant Cluster Survey (EDisCS) project (| White et al.ll2005l l. 
itself an ESO Large Programme (PI S. White, ID: 166.A- 
0162). The resulting spectroscopic sample of ERGS contains 
many tens of high redshift galaxies with secure redshifts 
jDouglas et al.ll2009t ). 

The structure of this paper is as follows. In section[5]we 
discuss the imaging data that was used as a basis for this 
work and briefly describe follow-up spectroscopic observa- 
tions. In section and we discuss the photometry and 
how the candidates were selected. We then discuss the final 
sample in section [5] and the estimated contamination frac- 
tion. The completeness of this sample is described in section 
[6l The morphology and observed UV luminosity function for 
the high redshift LBG candidates is considered in sections [7] 
and[S] 

We adopt the standard ACDM cosmology, i.e. a flat uni- 
verse with f^A = 0.7, Qm = 0.3 and Ho = 70km s~^ Mpc~\ 
All p hotometry was det ermined in the AB magnitude sys- 
tem (|Oke fc Gunnll 19831 ). 



2 OBSERVATIONS AND DATA 

In order to select a sample of objects containing 2; ~ 5 
LBGs all that is needed in principle is sufficiently-deep R 
and /—band imaging. Selecting a flux limited sample in / 
containing objects with an R — I colour redder than a partic- 
ular value (usually R—I > 1.3 or R—I > 1.5) should identify 
a sample of 2; ~ 5 LBGs with a completeness that depends on 
the statistical quality of the data. However, to cleanly select 
a sample of high redshift galaxy candidates without being 
swamped by contaminating cool Galactic stars and lower 
redshift galaxies, a wide range of multi-band deep imaging 
is required. 

2.1 Imaging 

The data set used in this study includes optical imaging 
using the V , R, I and 2 bands, near infrared imaging in 
the J and Ks bands and complementary high resolution /- 
band imaging from the Hubble Space Telescope (HST) of ten 
widely separated fields each of size ~ 40 arcmin^. The high 
Galactic latitudes of these fields result in minimal Galactic 
extinction effects and the original selection of the fields to 
observe z ~ 0.7 clusters produced an effectively random 
sampling of the sky at z 5, reducing the effects of cosmic 
variance. 

The m ajority of the imag ing data were taken as part 
of EDisCS (| White et al.ll2005h . a photometric and spectro- 
scopic survey of galaxy clusters selected from the Las Cam- 
panas Distant Clusters Survey (jGonzalez et al.ll200lh . The 
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observing programme involved deep optical imaging using 
F0RS2 at the VLT, near-infrared imaging using SOFI at 
the NTT, and high resolution imaging using the ACS cam- 
era on HST. The fields were observed with FORS2 in the V, 
R, and / bands for approximately 2 hours each, with a field 
of view after dithering of 6.5' x 6.5'. In the near-infrared, 
the fields were observed using SOFI on the NTT in the J- 
and ifs-bands for at least 300 and 360 minutes respectively. 
The effective field of view of the near IR observations was 
typically 5.4' x 4.2', consequently only around 50 per cent 
of each optical field was covered. This ground based imag- 
ing was cornpleme nted by 80 orbits of HST/ ACS imaging 
l|Desai et al.ll2007^ . Nine of the ten fields were observed for 
one orbit over the whole field and an additional four orbits 
in the central region using the F814W filter, with over 80 
per cent of each ground-based optical field being covered 
by HST data. With typical seeing conditions for the ground 
based observations between 0.5" and 0.8", the 2a depth in 
a 2" diameter aperture of each image was typically 28.1 in 
the V-hand, 27.9 in the R-hand, 27.1 in the /-band, 24.5 in 
the J-band and 23.7 in the isTs-band. Details can be found in 
Table [T] The depths of the images were calculated by mea- 
suring the noise properties of multiple randomly placed 2" 
apertures in background areas of the images. 

These images were reduced using standard techniques 
for bias and bad pixel removal and flat fielding. The software 
IMCAT findpeaks was used to detect local minima in images 
smoothed with a 1" Gaussian to create a catalogue of local 
sky measurements. These measurements were then fitted to 
a bi-cubic polynomial and subtracted from the unsmoothed 
image. The images were also aligned with known stars in 
the USNO catalogue to correct for camera distortions and 
the linear offset between the fields, preserving the surface 
brightness. Specific details of the data reduction producing 
the calibrated ima ges used in this study can be found in 
IWhite et al.l (|2005l l. 

2.1.1 Additional z-band Imaging 

The high redshift galaxies selected in this project are ob- 
served due to their bright rest-frame UV emission, redshifted 
to ~ 740nm at z ~ 5, arising from vigorous star formation 
within the galaxies. As young star forming galaxies, they 
are expected to have approximately flat continuum emission 
(/ly ^constant, an AB colour of zero) longward of Lyman a 
which can be probed in the observed z-band. In the absence 
of other effects, the I-z colour can also be an indication of 
a galaxy's redshift as the Lyman break moves through the 
/-band, (5.0 < z < 5.8). A high I-z colour would indicate 
very little flux in the /-band caused by the Lyman break 
located near the red edge of the /-band filter. Thus, the ad- 
dition of z-band imaging can improve photometric selection 
of high redshift galaxy candidates and the ability to study 
a candidate's properties. 

Each of the ten separate field s were observed for two 
hours in the z-band with F0RS2 (|Appenzefler et al.|[l998l ') 
using the z_Gunn filter and 2x2 pixel binning. The observa- 
tions were carried out in March and December of 2005 and 
February and March of 2007. For each field, forty-two ex- 
posures of two minutes each were taken at slightly different 
dither positions, similar to the technique used in the near 
infrared. Many short exposures are needed to minimise the 



increasing night sky emission at redder wavelengths. The 
high background is caused by line emission from the ex- 
cited levels of the hydroxyl radical OH~. This airglow can 
vary on timescales as short as five minutes so having many 
short exposures is beneficial. The fields have a mean depth 
of 26.18 AB magnitudes {2a limit in a 2" aperture) for an 84 
minute observation. The observations were taken with the 
same instrument as the other optical imaging and the expo- 
sures were planned to match the field of view of the existing 
images. The reduction of the data followed standard tech- 
niques using IRAF software. Bias subtraction, fiat-fielding, 
bad-pixel removal and photometric calibration using stan- 
dard star observations were all performed in the standard 
manner. 

Although the data were obtained with the same camera 
as that used to obtain the V, R and /—band datasets, in the 
time between the two sets of observations, the CCD chips 
were upgraded from SITE SI-424A backside thinned devices 
to MIT/LL devices. This upgrade improved the quantum 
efficiency of the instrument in the z-band, and almost com- 
pletely removed any fringing longward of 8000 A. It also 
changed the pixel scale of the data (from 0.2 to 0.125 arcsec 
per unbinned pixel), a change that influences how we carry 
out the photometry discussed section |3] 

2.2 Spectroscopy 

In addition to the z-band imaging, the ERGS program spec- 
troscopically observed a range of high redshift candidates 
and intermediate and low redshift contaminating objects. 
Although the spectroscopic follow-up will be presented in 
detail in a forthcoming paper, a brief summary is given as 
some of the results will be referred to in sections and [S] 

The high redshift galaxy candidates were foUowed-up 
spectrosco pically in a 10 hour s VLT/FORS2 program de- 
scribed in iDouglas et al.] (|2009l ). The 10 survey flelds were 
observed using 20 spectroscopic masks on VLT/FORS2 each 
containing 30 to 40 slitlets with a width of 1 arcsec and typ- 
ical length of 10 arcsec. Observations were carried out in 
service mode between December 2005 and May 2007 during 
dark time with seeing conditions better than 1 arcsec and 
clear or photometric conditions at an airmass less than two. 
Each mask was observed for 3.6 hours divided into expo- 
sures of 650 seconds nodded along the slit to avoid the effect 
of pattern noise or bad pixels and facilitate the best possi- 
ble sky subtraction. F0RS2 was used in spectroscopic mode 
using the R300I grating with the OG525 blocking filter to 
restrict the wavelength range to between 650-lOOOnm. The 
resulting pixel scale was 3.2A in the spectral direction and 
0.2 arcsec spatially and the spectral resolution was ~13A, 
measured from the width of the sky lines. 

The data were reduced fo llowing the same standard pro- 
cedures as those described in lLehnert &i Bremeij ()2003h and 
hence consistent with the data in the previous work. More 
details of the s pectroscopic da t a redu ction and analysis will 
be presented in lDouelas et al.l l)2009l l. 



3 PHOTOMETRY 

Object catalogues were initially created from the fully re- 
duced and calibrated images using the galaxy photometry 
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Table 1. Magnit ude limits fo r 2ct d etections in a 2" circular aperture. The moan weak Icnsing corrections for each field, as calculated 
from the maps of I Clowe et al. are shown in the last column. 



Field 


V 


seeing 


R 


seeing 


I 


seeing 


z 


seeing 


J 


seeing 




seeing 


Lensing mag 


J1037. 9-1243 


28.11 


0.55" 


28.04 


0.54" 


27.00 


0.56" 


26.17 


0.57" 


24.61 


0.82" 


23.93 


0.82" 


0.12 


J1040. 7-1155 


28.08 


0.65" 


27.95 


0.72" 


26.94 


0.62" 


26.12 


0.77" 


24.56 


0.70" 


23.49 


0.68" 


0.08 


J1054. 4-1146 


28.04 


0.67" 


28.07 


0.78" 


27.04 


0.72" 


26.03 


0.58" 


24.32 


0.83" 


23.53 


0.68" 


0.24 


J1064.7-1245 


28.07 


0.79" 


27.94 


0.77" 


27.24 


0.50" 


26.10 


0.62" 


24.45 


0.82" 


23.72 


0.78" 


0.26 


J1103. 7-1245 


28.05 


0.83" 


28.01 


0.75" 


27.19 


0.64" 


26.23 


0.55" 


24.38 


0.77" 


23.56 


0.63" 


0.20 


J1122. 9-1136 


28.12 


0.7l" 


28.00 


0.70" 


27.19 


0.65" 


26.20 


0.55" 


24.51 


0.79" 


23.59 


0.61" 


0.11 


J1138. 2-1133 


28.07 


0.58" 


27.86 


0.68" 


27.14 


0.60" 


26.14 


0.65" 


24.59 


0.82" 


23.69 


0.82" 


0.17 


J1216. 8-1201 


28.05 


0.68" 


27.92 


0.72" 


27.16 


0.60" 


26.23 


0.50" 


24.50 


0.76" 


23.58 


0.67" 


0.37 


J1227. 9-1138 


28.09 


0.73" 


27.89 


0.83" 


27.12 


0.74" 


26.27 


0.65" 


24.62 


0.96" 


23.67 


0.70" 


0.08 


J1354. 2-1230 


27.92 


0.70" 


27.96 


0.70" 


27.16 


0.66" 


26.35 


0.45" 


24.51 


0.90" 


23.98 


0.74" 


0.13 



package SExtractor version 2.4.4 (jBertin fc Arnouts|[l99^ . 
Magnitudes were determined using both 2" circular aper- 
tures and SExtractor defined MAG_AUTO apertures, which 
gave the best estimate of total magnitudes for individual ob- 
jects. Aperture corrections were determined for the circular 
apertures from the photometry of relatively isolated stars 
(identified as such using the HST imaging) using larger aper- 
tures. A comparison of the corrected circular aperture and 
MAG_AUTO aperture photometry for a range of unresolved 
sources showed agreement within statistical uncertainty. 

All high redshift candidates were expected to be unre- 
solved in the ground-based imagin g (they are found to have 
half-light radii of around 0.l"-0.3". lBremer et al]|2004 com- 
pared to typical seeing of 0.7"). The original intention was 
to apply the MAG_AUTO apertures to the other imaging 
bands with small corrections (usually a few hundredths of 
a magnitude) for differences in seeing in order to determine 
colours, or limits on colours. This required resampling the 
z, J and Ks-h&nA images to the same pixel-scale and as- 
trometry as that of the V, R and /-bands. Unfortunately 
the resampling significantly correlated the noise in the back- 
ground, particularly in the z-band. This had a significant ef- 
fect on the photometry of the faint {z > 25) LBG candidates 
(see figure [ij. Consequently, all subsequent photometry was 
carried out using 2" circular apertures on images which have 
not been resampled, taking into account the appropriate 
aperture corrections. This resampling did not affect the V , 
R and I images so we were able to additionally select ob- 
jects which had colours that fell into our selection criteria 
in the MAG_AUTO photometry for these bands. While not 
part of our main photometric sample discussed here, these 
were included on spectroscopic slit masks where there was 
available space. 

SExtractor was used in two image mode, assigning the 
/-band image as the detection image with which to define the 
photometric apertures for the three bluest images, V , R and 
/-band as these images were taken with the same pixel scale 
and central pointing. Objects were identified by a flux excess 
of more than 1.5a over at least four consecutive pixels. For 
the redder bands the /-band image was used as a detection 
image to deflne the centers of the apertures. These RA and 




Figure 1. The image on the left shows a portion of the fully 
reduced 2-band image where no rebinning has taken place. The 
image on the right is of the same portion of the reduced 2-band 
image rcbinncd to match the /-band image. The correlated pixel 
noise can be seen as a rectilinear pattern covering the background. 



Dec positions were used to place appropriately scaled 2" 
apertures on the z, J and Ks-hand images. 

SExtractor does not carry out photometry of an arbi- 
trary list of celestial coordinates. Consequently, this latter 
photometry was carried out using a modified version of the 
APER routine in the IDL astronomical library. In each case 
the photometry was carried out on images which had been 
background subtracted by SExtractor, using the APER rou- 
tine without any background subtraction, in order to mimic 
as closely as possible the photometry carried out in the bluer 
bands. The code was checked by comparing the resulting 
photometry of the /-band frame with that from SExtractor, 
both giving consistent results. 

The resulting catalogue containing V-, R-, /-, z-band 
photometry for all objects detected in the /-band, along 
with J and Ks photometry for those objects covered by 
the near -IR imaging, was th en corrected for Galactic ex- 
tinction (jSchlegel et al.lll998h producing the final photome- 
try catalogue used for high redshift candidate selection. The 
uncertainties in the measurements were assessed by plac- 
ing artificial galaxies of known size and flux randomly on 
the science image and then recovering them using the above 
methods. It was found that the measured errors approached 
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the theoretical Poissonian statistical error. Had an optimised 
aperture been used, such as the MAG_AUTO isophotal aper- 
tures used by SExtractor, the measured errors would have 
reproduced the statistical errors. In cases where objects were 
undetected in a single band, the 2a magnitude limit in a 2" 
diameter aperture was used to determine colours. 

These fields were originally chosen to study z ~ 0.8 
galaxy groups and clusters as part of the EDisCS project 
ij White et al.ll2005l ). Each field contains a clustered system 
at 0.6 < z < 0.8 with velocity dispersions generally be- 
tween 300 < o" < 900kms~^. One has a velocity disper- 
sion of ^ lOOOkms"^. These systems have a weak lensing 
effect across the fields and potentially a strong lensing ef- 
fec t over a sma ll area . We used the lensing models derived 
by I Clowe et al.l (|2006l l to determine the position-dependent 
amplification for sources at z ---^ 5 in order to determine 
the magnitudes of the sources in our (observed) flux limited 
catalogue in the absence of the intervening systems. These 
corrections were typically of ~ 0.1 magnitudes and we found 
no strongly-lensed z 5 candidates. The lensing effect was 
achromatic, changing the total magnitudes of objects but 
not their colours. 



4 LGB CANDIDATE SELECTION 

To take advantage of the near-IR imaging within this data 
set we use the following selection method, described step-by- 
step, to produce a robust sample of LBG candidates. The 
optical V-, R- and /-bands are used to select a flux limited 
sample with a continuum break which, if identified as the 
Lyman break, places the sources ai z > 4.8 (described in 
sub-section 14. ip . The near-IR imaging then becomes critical 
in identifying common low redshift stars and galaxies with 
colours red enough to satisfy the optical criteria. Having 
described such interlopers in sub-section 14. 2 1 sub-section l4.3l 
discusses the impact of the additional near-IR criteria. This 
leads to the final high redshift LBG candidate sample in 
section [S] which includes a discussion on the contamination 
fraction estimate from spectroscopic observations of a subset 
of the sample. 

4.1 Optical Selection Criteria 

In order to identify objects with the characteristic spectral 
break between the /- and i?-band of high redshift star form- 
ing galaxies, the selection criteria of J < 26.3 and R—I > 1.3 
were imposed to create a catalogue of objects with a signif- 
icantly large spectral break or a non-detection in the R- 
band. An additional condition of V > 27.5 was added as 
this band samples light around the Lyman limit at z ~ 5. 
As such, very little, if any, fiux is expected at these wave- 
lengths. Although applying this V-band cut will exclude a 
few per cent of the high redshift galaxies (typically those at 
redshifts 4.7 < z < 5.0 and magnitudes I < 25.5), it has 
the effect of decreasing the sample size by about 40 per cent 
and so substantially decreases the contamination fraction 
(see simulations in section [BJ. 

Including only those areas which are covered by all 
imaging bands (V-band to Ti'-band) and excluding regions 
near bright sources, a total of 319 objects were identified 
over the 10 survey fields covering an area of 275 arcmin^. 
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Figure 2. Colour modelling of high redshift star forming galaxies 
(blue) using a variety of stellar ages. A simple SED flat in f^ is 
represented by the dash-d ot line. The other three colour tracks are 
star forming galaxies from lMarastonI l l2005h with increasing stellar 
age of 10 Myrs (solid line), 100 Myrs (dotted line) and 400 Myrs 
(dashed line). All have had appropriate IGM absorption applied 
as described in the text. Al so shown are in termediate redshift 
elliptical galaxies (red) from iMarastonI l|2005l ) models, assuming 
a formation redshift of z=5 and a star formation e-folding time of 
T=0.5Gyr, an d cool, low mass stars with labels indicating subtype 
(purple) from iHawlev et al. I ll2002l ) templates, using appropriate 
FORS2 filter set for this study. Redshifts are noted for the high 
and intermediate redshift galaxies. 



Figure [5] shows the expected optical colour evolution 
of a high redshift LBG represented by the blue solid, 
dotted and dashed lines labelled from redshifts of 4.6 to 
5.6. The dot-dash line represents a SED which is fiat in 
and the solid, dashed an d dotted b l ue lin es are star 
forming galaxies taken from IMarastonI ()2005h integrated 
galaxy spectral synthesis models (updated PEGASE models 
(jPioc fc Rocca-Volmerang3ll997l i of the same age produce 
similar colours). Each track represents a different stellar age, 
10 Myrs (solid), 100 Myrs (dotted) and 400 Myrs (dashed), 
the oldest reasonable stellar age given the redshift range 
probed. 

All models have been adjusted for the effects of ab- 
sorption by intervening hydrogen cloud s in the intergalactic 
medium (IGM). We use the models of iMadau et al.l (|l999l ) 
for the evolution in the number density and column den- 
sity distribution of hydrogen clouds in the IGM to estimate 
the absorption as a function of redshift due to the Lyman- 
series and Lyman lim it. In common with other authors (e.g. 
iBouwens et al.|[2007^ ■ we modify the original Madau models 
to account for the hig her absorption seen in the spectra of 
high redshift quasars (jSongailal l2004l ) by requiring a more 
rapid evolution in the number density of absorbing clouds 
{i.e. {l+zf rather than (H-z)^ **^) beyond z=4.5. 

As the stellar age increases, the colours of the galaxies 
become redder. Although models with varying extinctions 
are not shown here to maintain the clarity of figure [21 there 
is a similar reddening effect with increasing extinctions. Ap- 
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plying the ICalzetti et al] (|2000l ') dust model to the galaxy 
models shown, the R — I colour reddens by ~0.2 magni- 
tudes and the I — z colour by ~0.1 magnitudes per 0.1 in- 
crement in E(B-V) from to 0.5. Given the single R — I 
colour cut used in this work we are able to include any I- 
band selected candidate regardless of it's intrinsic contin- 
uum slope and corresponding I — z colour. Results from the 
spectroscopic follow-up suggest that the majority of z ~ 5 
LBGs have intrinsically blue colours in the observed optical 
bands at wavelengths shorter than the Balmer break with 
little reddening, despite the fact that much redder objects 
could have been observed, justifying the use of relatively 
blue galaxy models in understanding the optical selection 
function. While ol der, redder objects do exist at z; ~ 5 (e.g. 
IVerma et aLll2007^ ■ they are a minority population. 

The single colour selection used here could have intro- 
duced a high number of candidates which are only detected 
in a single band potentially leading to a larger number of 
spurious sources caused by noise spikes. Our spectroscopy 
shows that this is not the case. Within the photometric se- 
lection there were 40 objects with a single band detection 
in the 7-band. Of these, 15 were observed spectroscopically 
down to the faintest /-band magnitudes and only one was 
not detected in the continua of the subsequent spectra. In 
the complete spectroscopic program (.Douglas ct al... 20091 ), 
which targeted fainter and bluer objects, an additional 10 
objects were only detected in the 7-band to 7 < 26.3 and all 
of which had continuum spectroscopically detected and are 
therefore real objects. 

These spectroscopic results suggest only one of 25 ob- 
served sources could be spurious giving an upper limit of 
4% of the single band detections for the fraction of spurious 
sources. At these faint magnitudes, a lack of spectral flux 
is not conclusive evidence that such a source is spurious. 
Thus, this upper limit could be a considerable overestimate. 
Assuming the spectroscopic results are representative there 
are, at most, only 2 objects within the sample predicted to 
be spurious and as such, this is not treated as a significant 
problem. 

The number of spurious sources was also checked by per- 
forming identical source detection routines on inverted im- 
ages. No spurious sources were identified using this method. 

4.2 Common Red Interlopers 

While a simple two or three-band colour cut can effi- 
ciently select a sample of distant LBGs, such a sample 
will inevitably include low mass Galactic stars and inter- 
mediate redshift pass ively evolving or reddened galaxies 
l|Stanwav et all l2008bl ) even without photometric scatter. 
By using the additional near-IR photometry we can sub- 
stantially decrease the number of these interlopers in our 
final sample. 

The first major potential interlopers that we need to 
consider are cool, low mass Galactic stars. To estimate ac- 
curate colours for the stellar interlopers applicable to the 
specific filter s et use d here, the stellar templates derived by 
iHawlev et al] (|2002l ) were processed through the FORS2 fil- 
ter set including detector response. Figure [5] shows the 7i — 7 
and 7 — 2 colours for M-type dwarfs. It shows that late M- 
types will be i ncluded in the samp le with a R — I > 1.35. 
Figure [3] shows iKnapp et all (|2004l ) near-IR stellar spectra 
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Figure 3. Colour modelling high redshift star forming galaxies, 
intermediate redshift ellipticals and low mass stars as described 
in figure [2] The dotted line represents the 2a colour limit of the 
I — J colour at the flux limit of the sample. 



convolved with the appropriate filter set. It shows that the 
J — Ka colour is relatively flat for M- and L-type dwarfs, 
however a larger range is observed in the I — J colour span- 
ning from 0.8 for the early M-type dwarfs to 3.5 for the late 
L-type dwarfs. 

The 2— a magnitude limits for our near-IR data are typ- 
ically J < 24.5 and Ks < 23.7. Given these near infrared 
depths late M-type and L-type dwarfs should be excluded 
from the high redshift sample through their 7 — J and J — Kg 
colours with limits of 7 — J > 1.8 (shown by the black dotted 
line in figure [3| and J — K > 0.8 for near-IR detections. Al- 
though early-type M-stars are not red enough to be detected 
in the near infrared imaging at the fiux limit of the sample, 
their relatively blue optical colours {R — I ^ 1) would ex- 
clude them from the high redshift selection. By contrast, 
true LBGs are expected to have relatively flat colo urs with 
7 - J < 1.8 and J - 7C < (jVerma et all l2007h and as 
such should not be detected in the near-IR imaging. As we 
note later, an advantage of using a single 7? — 7 colour cut is 
that it includes intrinsically red objects. However, as noted 
previously, the majority of spectroscopically confirmed high 
redshift galaxies had relatively blue continuum, supporting 
the use of the comparatively blue models. 

By incorporating the redder near-IR wavebands into the 
selection criteria we can exclude late M-type and L-type 
dwarf stars and early M-types from Ml to M3. The only 
stellar populations which have colours that can not be iso- 
lated using this imaging data set are M4 to M6 dwarf stars. 
However, the number density of M-stars is expected to de- 
crease with fainter magnitudes. 

The other major potential interloper of high redshift 
samples are the class of intermediate redshift elliptical galax- 
ies often referred to as EROs (extremely red objects). Such 
interlopers have a redshift between 0.5 and 2.0 and have 
redder optical to near-IR colours than the high redshift star 
bursting galaxies. Figure [2] shows the RIz colour evolution 
for a passively evolving intermediate redshift source. This 
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population of galaxies could enter the selection around a 
redshift ofz = 0.8-1.6, where the 4000 A break falls in the 
/-band. When combined with the underlying red colour of a 
passively evolving stellar population, this can lead to inter- 
mediate redshift objects with sufficiently red R — I colours 
falling into the simple colour selection. However, these ob- 
jects can again be identified and removed from the high 
redshift sample using near-IR imaging of sufficient depth. 
Different levels of dust could also redden these intermediate 
redshift galaxies in the optical encroaching them into the 
high redshift selection. Due to the /-band selection, dust 
reddening would lead to bright z, J and Ks colours lead- 
ing to such galaxies being rejected from the sample by the 
near-IR criteria in sub-section 14.31 

The intermediate redshift galaxy model from figure [2] is 
plotted in figure [3] which illustrates the I — J and J — Ka 
colour evolution. The model indicates that at a redshift of 
2 > 1.0 the galaxies would be detected in the near-infrared 
imaging with an / — J colour redder than the limits of the 
imaging even at the flux limit of / = 26.3. Intermediate 
galajcies at lower redshifts {z < 1) are in principle removed 
by the initial optical selection, however stochastic photomet- 
ric uncertainties may cause them to reenter the sample. 

Although the near-infrared data available to this study 
is not deep enough to detect the individual high redshift can- 
didates, the colours calculated above imply that many lower 
redshift interlopers (early and late M-type and L-type dwarf 
stars and ellipticals at z > 1) are detectable and can be 
identified. As such, any candidate with a clear near-infrared 
detection was removed from the high redshift galaxy sample. 
As figure 13] shows, galaxies at z ~ 5 are not expected to be 
detected in the near-IR imaging. Only galaxies with a high 
extinction, reddening the optical to near-IR colours, could 
expect to be detected. For the mean /-band magnitude of 
the sample, two magnitudes of reddening in the Ks-hand 
would be required for the object to be detected in the Ks - 
band imaging. Studies such as those of lVerma et al.l (|2007 ). 
which model the spectral energy distribution of similar high 
redshift galaxies, expect relatively low extinctions making 
such dusty objects rare. From Monte Carlo simulations us- 
ing the probability distributi ons of model s rathe r than the 
formal best fitting scenario, IVerma et all (|2007h found no 
objects with reddening greater than one magnitude and less 
than 10% of the formal best fitting models had sufficient red- 
dening to be detected the ERGS /fa-band imaging. Hence, 
removing all objects with a near-IR detection will reduce 
the interloper fraction while not significantly effecting the 
completeness of the high redshift sources given the rarity of 
sufficiently dusty sources. 

4.3 Near-IR selection criteria 

Using the near-IR imaging to identify common interlopers 
we exclude objects with J < 24.5 and/or Ks < 23.7 from 
the LEG sample. At close to the /-band limit these near- 
IR limits indicate colours incompatible with a high redshift 
identification (figure |4]). The effect of the near-IR colour cut 
is to exclude 66 objects. Within this sample 63 objects had 
I — J > 0.7 and J — K > 0.1 separating them from the 
theoretical colours of low extinction high redshift galaxies. 
The three remaining objects had blue J — K colours ap- 
proaching the high redshift colour tracks. However, all these 
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Figure 4. Colour modelling high redshift star forming galaxies, 
intermediate redshift ellipticals and low mass stars as described 
in figure [2] Ovorplotted are the colours of sources detected in the 
near-IR imaging and subsequently rejected from the high redshift 
sample. The colour error bars are shown for the confirmed high 
redshift galaxy. 



objects were bright (/ < 24.9) and unresolved in the HST 
imaging making them likely contaminants. Subsequently, all 
three were spectrosc opically confirmed to be at low redshift 
iDouglas et al.ll2009l ). 

During our programme of spectroscopic follow-up we 
observed a representative sample of rejected objects in or- 
der to confirm that these objects had a low probability of 
being genuine 2 ~ 5 LBGs. Of the 66 sources with near-IR 
detections, 24 were observed spectroscopically. Six spectra 
contained flux which was too faint to identify, one source 
was confirmed to be at high redshift (a continuum break 
galaxy at 2 = 5.45) and 17 were confirmed to be galac- 
tic stars or intermediate redshift galaxies (figure 1^. 95% of 
all sources detected in the near-IR imaging with sufficient 
spectral signal-to-noise were confirmed to be at low or in- 
termediate redshift. 

Although the LEG sample covers a relatively small 
range in /-band magnitudes, the interloper fraction cannot 
be assumed to be constant with flux. The /-band magni- 
tude distribution of the LEG candidates and the near-IR 
detected objects shown in figure [5] demonstrates the vary- 
ing interloper fraction as a function of magnitude. Although 
it is likely that some faint interlopers may not been identi- 
fied by the near-IR data given the depths of the images (see 
section [5^ . the fiat distribution observed suggests that the 
interlopers have the biggest statistical effect at bright mag- 
nitudes. This is at least in part due to the nature of the 
interloping samples. The fraction of stellar objects is ex- 
pected to decrease at fainter magnitudes (/ > 26) compared 
to brighter magnitudes due to the distribution of M-type 
dwarfs in the Galactic halo. This is also seen in the low level 
of contamination by stars in faint sam ples of HST-selected 
LEG candidates (jEunker et al.l l2004l l. Similarly, at these 
faint magnitudes the intermediate redshift elliptical galax- 
ies originate from the fiatter faint-end of their luminosity 
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Table 2. Source density per arcmin for each survey field. 
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Figure 5. Distribution of observed /-band magnitudes of final 
photometric sample (solid line) with near-IR detected sources re- 
jected and the sources removed from the sample because of near- 
IR detections (dotted line). 



function ratlier tlian tlie comparatively steeper exponential 
bright-end, assuming a typical Schechter function fit to the 
2 = 1 luminosity function. With the candidate high redshift 
galaxy number counts increasing towards fainter magnitudes 
this results in a lower fraction of intermediate redshift galax- 
ies at 7 > 26. 



5 THE Z ~ 5 GALAXY CANDIDATE SAMPLE 

The final photometric selection consisted of 253 objects in 
the refined sample, all with multiband photometry consis- 
tent with a redshift of 2 ~ 5 or higher, using the strict cri- 
teria discussed above. Figure shows the observed J-band 
fiux distribution for the refined sample of 253 objects and 
the 66 removed because of their near-IR detections. 

Significant field-to-field variation in the number of high 
redshift candidates was observed as shown in table (5] The 
range of number densities observed for the refined sample 
in individual fields can not be e xplained by cosmic vari - 
ance when using the estimator of iTrenti fc Stiavellil (|2008l ) 
suggesting observations of truly under and overdense fields. 
In contrast, the variation in the number density of near-IR 



detected sources can be explained by cosmic variance, esti- 
mated bv lTrenti fc Stiavellil (|2008l ) assuming a lower redshifl 
(z=l), with the exception of only two fields. 



5.1 Single versus two colour cut 

Due to the shapes of the filter bandpasses and the typical 
colours of low mass stars, a two colour selection, i.e. criteria 
in i? — / and I — z colours, is unable to efficiently select 
high redshift galaxies while eliminating M-type dwarf stars. 
Given this and the range oi I — z colours predicted for high 
redshift objects, a single R — I criteria was adopted. While 
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Figure 6. Colour-colour diagram showing spectroscopically con- 
firmed high redshift galaxies (crosses - line emitters, asterisks - 
continuum breaks) and confirmed lower redshift objects (galax- 
ies and stars where squares are objects undetected in nIR imag- 
ing and triangles are objects detected in nIR) compared to the 
field populations (points) . Arrows represent 2cr colour limits when 
there is no detection in corresponding band. 



the absence of an 7 — z colour cut may lead to a higher 
contamination fraction in a purely optically selected sample, 
the addition of near-IR selection criteria minimise this effect 
as discussed in section |4l 

The R — I and I — z colours of the spectroscopically 
observed sources within the photometric sample are shown 
in figure |H] The small dots show the range of colours seen 
in the catalogue of all objects in a single field. Over-plotted 
are the objects which were spectroscopically confirmed to be 
at high redshift (asterisks are galaxies identified using only 
their continuum break and crosses are galaxies identified 
with Lya line emission) and confirmed lower redshift galax- 
ies and stars with and without near-IR detections (triangles 
and squares respectively) . A number of sources, spectroscop- 
ically confirmed to be low redshift galaxies or galactic stars, 
have I — z colours bluer than that expected from the mod- 
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els in figure [5] This is most likely caused by a combination 
of photometric errors and different intrinsic properties of 
the sources to that assumed in the models. Figure [6] illus- 
trates that without the near-IR imaging many low redshift 
objects would not be distinguishable using only the optical 
colours. Also, had an I — z colour criteria been applied to 
eliminate lower redshift contaminants, many high redshift 
galaxies would have been excluded from the sample. 

A variety oi R — I and I — z colour cuts could be used to 
isolate the stellar interlopers from the LBG candidate sam- 
ple, e.g. I — z < 0.5. Adopting this additional colour cut 
should, in principle, exclude any stellar interlopers, however 
for the case of I — z < 0.5, 36% of the spectroscopically con- 
firmed high redshift sources would be rejected from the sam- 
ple and only three of the nine spectroscopically confirmed 
lower redshift interlopers would be removed. Thus, the ap- 
plication of such a colour cut for this combination of filters 
would significantly reduce the number of high redshift candi- 
dates biasing the sample to blue rest-frame UV continuum, 
while not significantly reducing the relative contamination. 

The additional near-IR selection criteria is an effective 
substitute for an I — z colour cut, removing common inter- 
lopers without influencing the high redshift LBG sample. 

5.2 Contamination Fraction from Spectroscopic 
Observations 

It is possible that for the optically-fainter objects we fail to 
reject contaminants because they are too faint to be detected 
in the near-IR data. At the flux limit of the sample there is a 
colour limit of / — J < 1.8 and I — Ks < 2.6 which does not 
exclude all possible low redshift contaminants even in the 
absence of photometric scatter. Lower redshift objects can 
also scatter into photometric selections through photometric 
error and extreme colours. 

Our spectroscopic programme was optimised to max- 
imise the number of spectroscopically-confirmed z ^ 5 ob- 
jects and had sufficient slits to sample robustly all objects 
which obeyed the optical selection. Consequently we were 
able to test if a given selection criteria improved the sam- 
ple. A third of all photometric candidates with a range of 
7-band fiux, colours and near-IR detections were observed. 
The spectroscopic identifications of this subsample can be 
used to estimate the fraction of contamination in the re- 
fined photometric high redshift sample. Although a spec- 
troscopic identification is harder to achieve for intrinsically 
fainter objects, figure [7] shows that our confirmed high red- 
shift LEGS are not biased to brighter J-band magnitudes 
and have a distribution similar to that of the refined sample 
as a whole (figure O. Contrary to what would normally be 
expected, fainter objects can be identified spectroscopically 
if they have a strong Lya emission line or a clear continuum 
break in a comparatively noise free region of the spectrum. 
These features can often make high redshift galaxies easier 
to identify than lower redshift objects without any emission 
lines. At low signal-to-noise, lower redshift objects can show 
essentially featureless continuum making any redshift deter- 
mination difficult, if not impossible. As such, any observed 
objects which can not be identified in their spectra due to 
low signal-to-noise, often up to 50% of the observed sam- 
ple, cannot be assumed to lie at high or low redshift; either 
scenario is possible. 
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Figure 7. Distribution of 7-band magnitudes for spectroscopi- 
cally confirmed high redshift galaxies (solid lino) and lower red- 
shift objects (galaxies and stars, dashed Une) within the raw pho- 
tometric sample. 



Of the 253 objects in the refined sample, 83 were ob- 
served on the spectroscopic masks. Within this subsample 
33 objects were confirmed to lie at z > 4.6, 9 were identi- 
fied as intermediate galaxies or stars and 41 were detected 
spectroscopically but the signal-to-noise was too low to al- 
locate a firm identification. Of the identified objects, 20% 
were at a redshift lower than targeted. Using the -R's-band 
imaging, the fraction of high and low redshift objects in the 
unidentified sample is estimated. 

While none of the individually confirmed high redshift 
galaxies were detected in Ks , by stacking 30 images we deter- 
mined that their average colour was I — Ks < 0.0. Similarly, 
a typical colour for the confirmed low redshift sources and 
the unidentified objects was obtained. The nine confirmed 
low redshift sources gave colour of I — Ks = 1.3. 201 uniden- 
tified objects in the refined sample, including those observed 
spectroscopically but resulted in low signal-to-noise, had an 
average colour of J — Ks — 0.3. Where there was no de- 
tection in the stacked i^s-band image the 2a limit for a 2" 
circular aperture was measured using background regions. 
Given the need for adequate background regions surround- 
ing the central stack to estimate the depth of the image, 
a small number of sources were not included as they were 
positioned close to the edge of the near-IR field. 

Assuming that the unidentified sample is a combination 
of the high {I - Ks < 0.0) and low redshift {I - Ks = 1.3) 
populations, the typical colours imply the unidentified sam- 
ple contains at least ~75% high redshift objects and less 
than ~25% low redshift objects. Combining these results 
with the spectroscopic identifications gives a contamina- 
tion fraction of ~20% for the refined sample. Applying the 
same technique to all optically selected objects including 
the near-IR detected objects, the contamination fraction is 
estimated to be around ~40%. The unidentified near-IR 



© RAS, MNRAS OOCCHIl] 



10 L. S. Douglas et al. 



detected sources have an average colour of 7 — Ks = 3.2 
(compared with a colour of 7 — Ks = 2.6 for the near-IR 
detected objects confirmed to be at low redshift) which im- 
plies a very high level of interloper within this subsample. 
The higher contamination rate estimated for the full sample 
clearly demonstrates the utility of the near-IR rejection of 
interlopers in addition to the purely optical selection. 

There have been several ot her spectrosc o pic s tudies 
of z ^ 5 LBGs, for example lAndo et all (|2007l l and 
IVanzella et al.l ()2009[ ). The differences in selection function 
and filter sets used in these studies makes a direct compari- 
son of the level of contamination difficult. One feature that 
all high redshift spectroscopic surveys share is the large frac- 
tion of unidentified sources due to insufficient signal-to-noise 
in the spectroscopy. This can be up to 50% of the observed 
sample. Such studies can appear reliable because most or all 
of the sp ectroscopica l ly ide ntified objects are at high red- 
shift, e.g. lAndo et all |2007l ) did not identify any lower red- 
shift objects spectroscopically. However, care must be taken 
when assuming the same contamination rates for the spec- 
troscopically unidentified sources and subsequently for the 
whole photometric sample, especially if only the brightest 
candidates have been targeted. 

Although it is difficult to compare contamination frac- 
tions of different samples due t o different selection, fil- 
ters and instrument combinations (jStanwav et al.|[2008bl l , it 
seem s that typical contamin ation rates are between 10% and 
20%. IStanTOv" et al.l l|2008bl l shows that with contamination 
fractions of this level, measurements such as the angular 
correlation function of photometrically-selected LBG sam- 
ples can be compromised because the expected contaminat- 
ing population can themselves have a clear correlation sig- 
nal. Also, the difference in optical-to-near-IR colour between 
contaminants and confirmed high redshift LBGs demon- 
strated here can also cause problems in estimating the typi- 
cal multi-wavelength spectral energy distributions of LBGs. 
If a population of contaminants individually-undetected in 
the near-IR are combined in a stacking analysis with true 
LBGs, they will redden the resulting SED, and artificially 
inflate the estimated age of the stellar population in a typical 
LBG or dust fraction. 



6 COMPLETENESS DETERMINATIONS 

The completeness of the selection in each held was esti- 
mated by carrying out two kinds of simulations. The first is 
a simulation of recoverability which depends on the quality 
and depth of the imaging. This involved injecting artificial 
sources with known properties into our photometry data in 
order to determine the efficiency of SExtractor in recovering 
them. The second simulation investigated the effects of the 
selection criteria and image quality on a model spectrum. 



6 . 1 Recoverability 

For each field, 300,000 artificial galaxies were generated cov- 
ering a range of magnitudes from 22.0 to 28.0 and a. R — I 
colour was assigned within the range 0.0 to 3.0. A surface 
brightness profile was chosen from a sample of four stellar 
profiles measured directly from the images to reflect the see- 
ing conditions (the high redshift candidates are unresolved 



in ground-based data), and a selection of HST/ACS images 
of two or more galaxies seen at small angular separations se- 
lected in the GOODS data of the Chandra Deep Field South 
(CDFS). The high resolution multiple systems were con- 
volved with a Gaussian profile to match the ground-based 
seeing conditions and were chosen with the same frequency 
as seen in the CDFS (~20%) to mimic this population. Us- 
ing the selected magnitude, colour and profile, a 'simulated' 
galaxy was created for the 7-band and 7?-band images and 
placed at random coordinates within the field excluding pre- 
viously masked areas near bright objects used in the pho- 
tometric selection. By placing these galaxies directly onto 
the science image, the noise properties of the image were 
added to the simulated galaxy. The process was repeated 
one thousand times for each field as no more than 300 sim- 
ulated galaxies were placed on any one image to prevent 
mutual overcrowding affecting the results. 

Each new image with the simulated galaxies was pro- 
cessed through SExtractor in the same way as the origi- 
nal science data, creating catalogues containing the simu- 
lated galaxies as observed on the images. After these were 
matched with the original galaxy information on magnitude, 
colour and position, a catalogue was created containing the 
known simulated magnitudes and the recovered SExtractor 
magnitudes of each fake galaxy. 

The high redshift galaxy candidates were selected on 
two main criteria, a magnitude limit and a colour cut. The 
recoverability of the galaxies using the high redshift selec- 
tion of 7 < 26.3 and R — I> 1.3 was measured for a range of 
colours and magnitudes and is shown in the contour plots of 
figurelHl Eight of the ten fields show comparable results, with 
a completeness of 25% at the 7-band flux limit. However two 
fields had consistently lower completeness which was caused 
by the two worst seeing conditions for the 7-band observa- 
tions, 0.73" compared to an average of 0.6" over the other 
eight fields. This spread the received flux over a larger area 
causing the objects to drop below the surface brightness 
limit. If the objects were detected, the photometry, made 
fainter by the seeing conditions, is corrected by the appro- 
priate aperture corrections. Figure |8] shows an average of the 
ten survey fields, with the simulated 7-band magnitude and 
R — I colour along the x and y axes respectively. The con- 
tours are placed at 20%, 40%, 60%, 80% and 90% of inserted 
galaxies recovered, using the colour selection of 7 < 26.3 and 
R — I > 1.3. Figure |5] shows that the completeness of object 
recovery is generally independent of colour with the results 
reflecting the increasing photometric errors at fainter magni- 
tudes. The maximum completeness never rises to more than 
~ 95% due to object confusion despite areas near bright 
objects being masked. 

Figure [8] implies that only 60% of galaxies with an in- 
trinsic R~ I colour of 1.3 would be included in the sample. 
This is not surprising as only ~50% of objects would be ex- 
pected to be included in the sample at the colour cut. Even a 
small photometric error in the 7? or 7-band would cause the 
colour to become too blue. The applied R — I colour criterion 
was c hosen from the previous work of iLehnert fc Brerneil 
(|2003l ') who found a colour cut of 7? — 7 > 1.5 selects galax- 
ies at z > 4.8 using th e same F0RS2 filters used in thi s 
work and models from iFioc fc Rocca-Volmerang3 l)l997h . 
By adopting a bluer colour cut, completeness at this selec- 
tion colour is much higher. Although this could introduce a 
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Figure 8. Contours illustrating the success rate of recovering 
galaxies within the high rcdshift colour selection. The levels shown 
are 20%, 40%, 60%, 80% and 90%. The x-axis is the original I- 
band magnitude of the 'fake' galaxy inserted into the images and 
the y-axis is the original colour assigned to this 'fake' galaxy. The 
contours show the percentage recovery rate of these 'fake' galaxies 
using the same method as the high redshift selection. It is shown 
that the completeness of object recovery is generally independent 
of colour with the results reflecting the increasing photometric 
errors at fainter magnitudes. 



higher contamination fraction, the additional near-IR imag- 
ing should minimise this. 

Also, the objects that were recovered with intrinsic 
colours bluer than our cut represent a population of ob- 
jects that can contaminate the sample, rather than one that 
affects the sample completeness. These objects have their 
measured colours reddened by the statistical errors on the 
photometry. At J = 26 a substantial fraction (~ 20%) of ob- 
jects with intrinsic i? — / = 1.1 can be included in a sample 
with an observed cut of i? — / > 1.3 given the photometric 
properties of our source data. Such objects are likely to be at 
z < 4 and could be a substantial contaminating population 
depending upon their luminosity function. In reality, such 
objects would often have significant V-band flux or near-IR 
detections which would rule them out of the high redshift 
candidate sample. 



6.2 Effect of Selection Criteria 

The second simulation determined the completeness of our 
final sample given the interplay of photometric cuts used 
to reject contaminants and the statistics of the photometry 
data, as a function of redshift. An artificial catalogue of 10** 
sources was generated wit h a J-band magnitude distribution 
determined by the z = 5 iBouwens et al.l (j2007[ ) luminosity 
function. Each galaxy was assigned a redshift (in 0.01 red- 
shift bins) which determined its /-band magnitude and R—I 
colour assuming a flat spectrum in Fi, and accounting for 
IGM absorption using the modified Madau perscription as 



Figure 9. The fraction of galaxies recovered as a function of 
rcdshift with an intrinsic I < 26.3, R - I > 1.3 and V > 27.5 
given the photometric errors of the imaging data. The luminosity 
function use d to generate the pred icted number counts was that 
measured in lBouwens et all ll2007t) for z ~ 5. 



a function of redshift (described in section |4T|) . The V-, R- 
and /-band magnitudes of the artificial sources were then in- 
dependently perturbed to match the typical noise properties 
of the imaging data and the typical weak lensing character- 
istics from the foreground clusters. The fraction of objects 
which were then recovered using the previous z ~ 5 selection 
criteria (/ < 26.3, R — I > 1.3 and V > 27.5) as a function of 
redshift is shown in figure[9] In total, around 80% of galaxies 
at 5.0 < z < 6.0 are selected using the photometric colour 
criteria of this work but completeness falls off rapidly below 
z = 5. 



7 MORPHOLOGY 

The morphologies of the high redshift candidates were stud- 
ied using high resolution images from the Hubble Space Tele- 
scopes using the ACS camera and F814W filter which cover 
nine out of the ten survey fields. Each field was observed 
for 1 orbit with the central regions observed for a total of 5 
orbits. 

Catalogues from the HST/ACS images were created us- 
ing SExtractor in order to obtain half-light radii for the po- 
tential high redshift galaxies. Only sources with the deepest, 
5 orbit depth of data are discussed. 

The majority of high redshift candidates in the refined 
sample were found to be resolved with individual peaky com- 
ponents. They have half-light radii between 0.1" and 0.26" 
(figure [T0| which corresponds to a range of 0.3-1.6 kpc at 
a redshift of z ~ 5 . with a mean value of 0.14" or 0.8 kpc 
in agreement with Bremer et al. (|2004[). th e purely space- 
based sample of IConselice fc Arno ld' l200^) and the higher 
redshift galaxies of Bouwens et al.l (^2006,) . A subset have for- 
mally unresolved radii (~0.06"). However, these objects are 
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Figure 10. Distribution of half-light radii, in arcseconds, of the 
refined high redshift sample with the deepest HST data. Radii 
below 0.07" are unresolved. 



generally faint with limited S/N that could be effecting the 
measurement and fitting procedures. While some of these 
could potentially be stars, their colours do not support this 
hypothesi s. It is also poss i ble to select AGN such as that de- 
scribed in lDouglas et al.) l)2007h which could be unresolved 
in the HST imaging. 

Over the 9 fields, 31 out of 95 (33%) candidates with 
deep HST data show disturbed or asymmetric morphology 
or are part of a multiple system on scales from 0.2" to 1.2" or 
l.Skpc to 7.5kpc. Examples of multiple systems, which are 
not resolved in ground based imaging, are shown in figure fTTI 
Each individual component has a half-light radius consistent 
with the the isolated objects. It is not clear whether these are 
most likely to be multiple interacting systems or UV-bright 
knots in an underlying larger system. 

Of the multiple systems, 10 candidates are close pairs 
and 4 are part of a larger collection of sources with a mean 
separation of 0.5". Objects with separation distances of more 
than 0.8" are barely resolved in the ground-based imaging 
and extend with the same orientation. The fractions of ob- 
jects o bserved with multiple members (15%) agrees with the 
work of lConseUce fc Arnold! (j2009l l in the Hubble Ultra Deep 
Field (HUDF) where ~19% of their V-drop sample were 
found to be in pairs. With this much deeper data, the pos- 
sibility that the observed galaxy pairs are a larger object 
with UV bright peaks is less. As the sample fractions show- 
ing galaxies pairs for the deep HUDF sample and the sample 
presented here are similar, it is likely that these are the true 
pair fractions for each sample. 

The 17 candidates which show disturbed or asymmet- 
ric morphology were identified visually. Even in the deepest 
available HST imaging the objects are small and faint mak- 
ing any detailed measurement of their structure challeng- 
ing. F or a similar sample in the HUDF. fConselice fc Arnold! 
l|2009h observed the same fraction exhibiting asymmetric 
morphologies. With the much deeper observations of the 




Figure 11. Examples of high redshift galaxy candidates with 
multiple components or disturbed morphology visible in the high 
resolution HST/ACS F814W imaging. Each image segment is 
2.5" X 2.5". 



HUDF, more precise measurements of the structure and po- 
tential asymmetry of the high redshift galaxy candidates 
was possible. Although only visual classification was used 
for the ERGS sample, the agreement of asymmetric sample 
fractions suggest that such a classification is reliable. It also 
suggests that the fraction of multiple of asymmetric systems 
does not change with UV luminosity over the range probed 
by the two studies. 



8 THE UV LUMINOSITY FUNCTION 

One apparently straightforward measurement of the prop- 
erties of z ~ 5 LBGs that can be made with a photomet- 
ric sample is the rest-frame observed UV luminosity func- 
tion. For this to be reliable and meaningful, the contamina- 
tion and completeness of the sample must be understood. 
While our reliable photometric sample is constrained to 
brig ht magnitudes in com parison to some published data 
(e.g. !Bouwens et al.l !2007! l , it can be used to explore the 
brighter end of the luminosity function. 

In order to calculate a luminosity function which rep- 
resents the observed luminosity distribution of galaxies, the 
fol lowing method, which is essentially the same as that used 
bv !Bouwens et al] (|2007! ). was used. Monte carlo simulations 
were used to create artifical populations of LBGs across the 
redshift interval probed photometrically, with corrections for 
photometric errors, completeness and weak lensing. After an 
identical photometric selection, the resulting artifical LBG 
candidates represents what would be observed given the in- 
tial luminosity function assumed. When compared to the 
real, observed LBG candidates, a best-fitting intrinsic lumi- 
nosity funtion can be found. 

A range of Schechter luminosity functions (defined by 
the characteristic magnitude M* , the faint end slope a and 
normalisation (jf) were generated across a broad spectrum 
of parameters, from —19.65 to —21.45 in M'Ijy, faint end 
slopes, a, between —1.40 and —1.95 and cj>* fitted freely, 
and were compared to the observed number counts of the 
2 ~ 5 galaxy candidates. 

For each proposed luminosity function the following 
steps were used to create a catalogue of galaxies which re- 
flects the assumed luminosity function and the properties of 
the observed fields. Within 0.01 redshift bins from 2 = 4 to 
z = 6.1, the luminosity distance, luminosity function and 
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IGM absorption were used to determine the apparent mag- 
nitude distribution and number density of objects scaled ac- 
cording to the volume within the redshift slice (per arcmin^). 
The colours of the objects in this redshift bin were calculated 
according to the FORS2 instrumental response and filter 
profile s assuming th e galax ies were flat in F„ and incorpo- 
rating iMadau et alj (|f 999l ) IGM absorption with a higher 
density of absorbing systems at high redshift as discussed in 
section im The galaxy catalogues for each redshift bin were 
combined to create a sample with a magnitude, colour and 
redshift distribution which reflects the proposed luminosity 
function. 

To simulate the properties of the survey fields, each 
galaxy in the artificial catalogue had colours and magnitudes 
perturbed by the photometric errors and typical weak lens- 
ing correction {e.g. if 20% of the survey area was lensed by 
0.1 magnitudes, 20% of the catalogue was randomly selected 
and the model magnitudes were brightened.) Using this arti- 
ficial catalogue, the galaxy candidates were reselected using 
the original selection criteria (/ < 26.3, R — I > 1.3 and 
V > 27.5) to produce the predicted magnitude distribution 
for the ERGS survey for the assumed luminosity function 
with magnitude dependent completeness corrections (as de- 
scribed in section [6l in addition to the photometric errors, 
caused by the ERGs image quality, and weak lensing correc- 
tions. 

Having repeated this procedure for each combination of 
luminosity function parameters, the predicted and observed 
number counts were compared using statistics across the 
range of M* and a, allowing the normalisation, 0* to vary. 
The resulting best fitting observed luminosity function is 
shown in figure [T^ As this method does not formally fit 0*, 
the correction for contamination is applied to the normali- 
sation after the fitting procedure. 

As figures [T^ and [T3] show, our results for the re- 
fined sample, a re ge nerally in agreement with those of 
iBouwens et al.l ()2007h . That result was derived from ex- 
tremely deep multi-band HST imaging of a comparatively 
small area. The fidelity of the data means that it pro- 
duces an intrinsically cleaner sample with less contamina- 
tion than typical ground-based studies. We determined a 
formal best-fitting M* = -20.96 ± 0.2 at 1500A 0.3 m agni- 
tud es fainter than tha,t found bvlBouwens et al.l (l2007h (see 
also l Oesch et al.ll2007l : iBeckwith et al.ll2006l : lYoshida et'al] 
\2OOt , for additional space and ground-based studies) with 
Q = —1.4. Given that the best-fitting M* falls into one of the 
faintest bins used for model fitting, the data set in this paper 
can not constrain the faint end slope (figure [T3|) . When a was 
set to -1.6 the best fitting characteristic magnitude was 
M* = -20.9±0.2 and 0* = 1.1±0.2 x 10"^ Mpc"^ agreeing, 
within the errors, with previous studies. As we are observing 
z — 5 LBGs using an /-band filter at 7680A the measured 
luminosity functio n is at 1250A compa red to I6OOA lumi- 
nosity function in iBouwens et al] (|2007h . Such a difference 
in wavelength would result in difi'erent luminosity function 
parameters at lower redshift, due to the spectral slope of 
the source. As high redshift LBGs have relatively flat con- 
tinuum, particularly between 1250A and I6OOA changes in 
the luminosity function parameters are expect to be small. 

Assuming the luminosity function fitted to the refined 
sample defines the shape of the relationship, the normalisa- 
tion, as defined by (p* , can be adjusted to include the relia- 
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Figure 12. The observed galaxy number counts across the 10 
survey fields compared to the best fitting Schechter lum i nosity 
function (dotted line) with a = —1.6 fixed. [Steidel et al.l lll999l ) 
2 ~ 4 luminosity function applied to z = 5 assuming no evolution 
and given the imaging properties and selection of this study is 
shown by the dashed line. The faintest /-band magnitude bin 
was not included in the fitting procedure. The errors shown arc 
the Poissonian errors of the number of objects in each bin. 




Figure 13. space illustrating the fitting of Schechter functions 
to the observed galaxy number counts. The contours levels are 
90%, 95% probabilities. With such a faint M* relative to the depth 
of the sample, the slope of the faint-end, a, can not be constrained 
with this data. 
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bility of this sample (-^80%, see section . The adjusted 
normalisation gives (f>* = 0.9 ± 0.2 X 10"^ Mpc"^ where the 
fraction of objects at high redshift is assumed to be con- 
stant with apparent magnitude. However, figure [S] suggests 
that contamination is a more significant problem at brighter 
magnitudes with a high fraction of lower redshift objects 
compared to high redshift candidates. A magnitude depen- 
dant contamination correction, with a larger correction in 
the brightest bins, would change the shape of the luminos- 
ity function leading to a fainter M* . Even with the extensive 
spectroscopic foUowup carried out in the ERGS project, the 
number of confirmed high redshift galaxies is not sufficient 
to study the reliability of the photometric sample as a func- 
tion of apparent magnitude. 

For consist ency with other work, we have used a 
ISchechteil (| 19761 ) luminosity function form for fitting. We 
note that the suitability of an analytic form derived from the 
expected form of galaxy mass functions is questionable for 
these intense starburst events. The UV emission arising from 
these LBGs is highly stochastic, tracing as it does short lived 
intense star formatio n events. The char acteristic timescale 
for these is ~ 30Myr (jVerma et al.ll2007l ) in comparison to a 
survey window of 250 Myr, i.e. the measured UV luminosity 
function is only capturing a fraction of the overall LBG pop- 
ulation. The UV luminosity function is therefore likely to be 
weakly related to mass distribution of star forming galaxies 
at these redshifts and therefore there is little a priori rea- 
son to expect it to follow a Schechter function. Indeed, it is 
plausible that the UV luminosity function is shaped by the 
time-dependent decay of UV emission from these short-lived 
starbursts, in effect the shape of the luminosity function mir- 
rors the UV emission (and therefore star formation) history 
of a typical 2; ~ 5 LBG starburst. Although our data can 
be fit by a Schechter function, it can be equally well fit by 
a simple power-law, N(L)(xL^^'^. The main issue associated 
with determining the detailed UV luminosity function from 
bright to faint magnitudes at this redshift is the limited vol- 
ume probed to the faintest levels. This necessitates combin- 
ing together the number counts from at least two separate 
samples covering the bright and faint ends of the function. 
As these come from different volumes, the normalisation of 
the two ends can vary due to field-to-field variation caused 
by cosmic variance, or by magnitude-dependent contamina- 
tion levels within the samples. Indeed, an intrinsic power 
law extending from bright to faint levels could manifest an 
apparent break if the bright and faint regimes were deter- 
mined from separate samples and the bright subsample was 
normalised high due to cosmic variance or a higher level of 
magnitude-dependent contamination. 

8.1 Sources of Reionization 

iLehnert fc Bremed l|2003h claimed that the number of 2 > 5 
sources, selected in a similar manner to this work, was less 
than that expected from the 2 = 4 luminosity function of 
Lyman Break galaxies (about 1/3 the number of sources 
for / < 26.3), assuming no evolution between the epochs. 
Although their data was taken with the same instrument 
as used in this work, their conclusion was drawn from only 
10% of the sky area of this survey. Nevertheless, our average 
number counts over 10 fields also implies a similar lack of 
bright sources compared to that expected if there was no 



evolution in the luminosity function between 2 = 5 and 2 = 
4. 

Figure [12] compares the observ ed galaxy number co unts 
(crosses) with that predicted using ISteidel et al] |l993) 2 = 
4 luminosity function down to M* . It can be seen that the 
observed number counts are a factor of ~3 less than would 
be expected if there is no evolution in the luminosity func- 
tion between 2 ~ 5 and 2 4, a deficit which increases with 
luminosity. Although the epoch of reionization has ended by 
2^5, the Universe still requires a sufficient density of UV 
photons to counteract the process of recombination and keep 
the hydrogen ionised. Even without the removal of contam- 
inants, there is a lack of galaxies providing ionising photons 
at these magnitudes. 

The equation for the density of star-f ormation required 
for reionization from iMadau et al.l l|l999h , modified to the 
cosmology assumed in this study, was used to estimate the 
star-formation density required to maintain the ionisation 
state of hydrogen at a redshift of 5.1, the mean redshift of the 
spectroscopically confirmed galaxies within the photometric 
sample. The concentration factor of hydrogen was assumed 
to be 30 and the photon escape fraction used was /esc = 
0.14 (|Shaplev et aLlliood ). however these values are not well 
constrained and can vary largely. The total luminosity of 
the sample was found by converting all apparent magnitudes 
assuming the mean redshift of the confirmed high redshift 
galaxies within the sample for all objects. This was then 
co nverted to a s tar-fo rmation rate using the relation given 
in lMadau (|l998h . 

It was found that the star-formation density of the 
refined sample is a factor of ~4 lower than that neces- 
sary to maintain the ionisation of hydrogen at redshift ~5. 
Even when the whole photometric sample is used, with- 
out the removal of likely contaminants, the star-formation 
density is still nearly 3 times lower than required. These 
results imply that the majority of ionising photons origi- 
nated from fainter sources below th e observed flux limit, 
(/ > 26.3; iLehnert fc Bremej |2003| ) even after the Uni- 
verse has fully reionized. Anothe r potential source of ion- 
ising photons are AGN. However, [b remer et al.l lj2004h and 
iDouglas et al.l (|2007l ') have shown that the density of AGN 
at these redshifts is very low and as such can not be a large 
contributor to the overall photon budget of the Universe at 
these redshifts. 



9 CONCLUSIONS 

Through a photometric colour selection we have produced 
a sample of 253 high redshift Lyman Break Galaxy can- 
didates at 2 5 drawn from 10 widely-separated fields 
with V, R, I , z, J, Ks photometry covering 275 arcmin^. Us- 
ing simulations and a comparatively large programme of 
follow-up spectroscopy we have been able to determine the 
completeness and reliability of the sample. The sample of 
253 sources is approximately 80 per cent reliable {i.e. has 
a 20 per cent contamination rate). Even with multiple 
photometric bands used to select objects, the relatively ex- 
treme colours of dropout galaxies make them vulnerable to 
systematic effects in photometry of individual objects. Con- 
sequently, statistical results derived from large samples of 
photometrically-selected objects may well be prone to bias 
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or error because of contaminatio n within the samples. A s has 
been pointed out elsewhere (e.g. IStanwav et al.|[2008bl ) , con- 
tamination rates of only 15 per cent can significantly skew 
some statistical results. 

A large fraction of these galaxies have multiple com- 
ponents on scales probed by HST imaging. It is currently 
unclear whether the components are separate galaxies, or 
UV luminous regions embedded in a larger, darker under- 
lying galaxy but deeper HUDF studies suggest we are ob- 
serving multiple galaxy systems. Previous work has shown 
that LB Gs appear to decrease in linear size with increasing 
redshift (|b ouwens et al.l 120041 ') . This applies to individual 
components of multiple systems, but clearly not necessarily 
the systems themselves. 

Contamination is a potentially serious issue in con- 
straining the 2 ~ 5 luminosity function. Having taken into 
account our detailed reliability analysis, we confirm a decline 
of a factor of ~ 3 in the bright end of the luminosity function 
(to M*) from z = 4 to z = 5. Assuming a Schechter fit to 
the UV luminosity function at z ~ 5, we obtain values for 
M^v = -20.9 ± 0.2 and <;/>* = 0.9 ± 0.2 x 10"^ Mpc"^ (with 
a magnitude in dependent correction for completeness), in 
agreement with lBouwens et all (|2007l l and others. However, 
we question the basis for the use of a Schechter function. 
This is a proxy for a mass function, whereas the UV lu- 
minosity of LBGs arises from highly stochastic, short-lived 
starburst phases and should not necessarily correlate with 
the mass of the underlying system. 
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